Stat 461 – Applied Probability Models Some Lecture Notes

These notes are based on the two lectures1 I gave in Stat 461, Applied Probability Models, as a substitute for Professor Cheng Ouyang. First, we cover the long-run behavior of Markov chains, in particular, the convergence to a stationary distribution. Second, an application to Markov chain Monte Carlo computation is given, namely, the construction and justification of the Metropolis–Hastings algorithm. Long-run behavior of Markov chains Recall, a Markov chain is a stochastic process, i.e., a sequence of random variables, with the property that the conditional distribution of Xn+1, given the past X1, . . . , Xn, depends only on the most recent state Xn. Therefore, the process {Xn : n ≥ 0}, on the finite state space {0, 1, . . . , N}, can be completely characterized by the pair (μ, P ), where μ = (μ0, μ1, . . . , μN) is the initial distribution, and P is the transition distribution in the form of a (N + 1)× (N + 1) matrix, with Pij denoting P(Xn = j | Xn−1 = i), for any n. Then P , the matrix power, with (i, j) entry P (n) ij denoting P(Xn = j | X0 = i), and the goal here is to study the limiting behavior of this quantity as n→∞. Question. What can be said about P (n) ij as n→∞? Limit exists, unique, etc? As a first step towards understanding the limit of the transition distributions, we have the following fact, a consequence of the celebrated Perron–Frobenius theorem, often discussed in courses on linear algebra: if P is “regular,” then the limit πj = lim n→∞ P (n) ij exists and does not depend on i. 3 These lectures are based, in part, on Pinsky and Karlin’s Introduction to Stochastic Modeling. http://en.wikipedia.org/wiki/PerronFrobenius_theorem That the limit πj does not depend on the initial state i has some intuition. For example, in calculus, the limit of a sequence cannot, by definition, depend on where the sequence starts. Moreover, by the Markov property, the chain does not hold on to the past, so the initial state-independence is quite reasonable. On the other hand, the fact that there is no dependence on the starting point is a practically important property.